Multispectral sample imaging

ABSTRACT

The disclosure features methods that include exposing a biological sample to illumination light and measuring light emission from the sample to obtain N sample images, where each sample image corresponds to a different combination of a wavelength band of the illumination light and one or more wavelength bands of the light emission, where the one or more wavelength bands of the light emission define a wavelength range, and where N&gt;1, and exposing the sample to illumination light in a background excitation band and measuring light emission from the sample in a background spectral band to obtain a background image of the sample, where the background spectral band corresponds to a wavelength within the wavelength range.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/666,697, filed on May 3, 2018, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to multispectral imaging of biological samples,in particular by fluorescence microscopy.

BACKGROUND

Fluorescence microscopy techniques are used in pathology to provideinformation about disease and patient response, to advance humanunderstanding in research settings, and to guide individual patienttreatment in clinical settings, Most fluorescent microscopy makes use ofdyes that are engineered or selected to provide a strong fluorescencesignal with known excitation and emission properties.Immuno-fluorescence labeling techniques allow individual epitopes in asample to be targeted with specificity, so that proteins, antibodies,nucleic acids, and other tissue and cellular components can be examinedeven when present in relatively small quantities.

SUMMARY

In multispectral imaging, multiple dyes or stains are applied to asample, and then the sample is imaged over a broad range of wavelengths,yielding a combined set of spatial and spectral measurements. The dyesor stains can be applied to the sample such that each stain bindsspecifically to a sample component of interest. Contributions of each ofthe sample components are separated by analyzing the set of spatial andspectral measurements to determine the quantity and spatial distributionof each of the components of interest within the sample. Certainfluorescent dyes have relatively well-defined excitation bands andemission bands. Accordingly, for some samples, it may be possible tochoose several dyes—typically 3 or 4—having emission bands that aredistinct enough so that they can be detected via spectral filteringwithout excessive cross-talk among the emission bands.

Tissue samples exhibit some degree of endogenous fluorescence response,which is termed autofluorescence. Autofluorescence may be stronger informalin-fixed paraffin-embedded (FFPE) samples than in fresh-frozensamples, perhaps as a consequence of the fixation chemicals or thefixation process. Unlike engineered dyes, autofluorescence is a naturalphenomenon and does not have well-defined excitation and emission bands.To the contrary, autofluorescence has a broad emission spanning much ofthe visible range, making its emission hard to distinguish from thefluorescence emission due to dyes that are specifically bound tocomponents of interest in the sample.

This disclosure features methods and systems for measuringautofluorescence in samples to which one or more dyes or stains havebeen applied. In general, for a sample to which multiple dyes or stainshave been applied, fluorescence emission is measured in a number ofspectral wavelength bands, each of which may correspond to fluorescenceemission largely from only a single one of the dyes or stains.Fluorescence emission is also measured in an emission band that does notcorrespond to significant emission from any of the applied dyes orstains. Emission in this band largely corresponds to autofluorescencefrom the sample. Based on information derived from measurement offluorescence emission in this band, autofluorescence throughout thesample can be quantified. The quantified autofluorescence can then beused to correct fluorescence emission measurements for some or all ofthe dyes or stains, and further used to quantitatively measure some orall of the dyes or stains at different locations within the sample.

In one aspect, the disclosure features methods that include: exposing abiological sample to illumination light and measuring light emissionfrom the sample to obtain N sample images, where each sample imagecorresponds to a different combination of a wavelength band of theillumination light and one or more wavelength bands of the lightemission, where the one or more wavelength bands of the light emissiondefine a wavelength range, and where N>1; and exposing the sample toillumination light in a background excitation band and measuring lightemission from the sample in a background spectral band to obtain abackground image of the sample, where the background spectral bandcorresponds to a wavelength within the wavelength range, and where foreach of one or more non-endogenous spectral contributors in the sampleexposed to the illumination light in the background excitation band, aspectral emission intensity at each wavelength within the backgroundspectral band is 10% or less of a maximum measured spectral emissionintensity of the non-endogenous spectral contributor followingexcitation of the sample in each of the wavelength bands of theillumination light and the background excitation band.

Embodiments of the methods can include any one or more of the followingfeatures.

The methods can include obtaining an autofluorescence image of thesample from the background image. For each of one or more non-endogenousspectral contributors in the sample exposed to the illumination light inthe background excitation band, the spectral emission intensity at eachwavelength within the background spectral band can be 4% or less (e.g.,2% or less) of the maximum measured spectral emission intensity of thenon-endogenous spectral contributor following excitation of the samplein each of the wavelength bands of the illumination light and thebackground excitation band. The background spectral band can include adistribution of wavelengths having a full width at half maximum (FWHM)spectral width Δλ, and a center wavelength λ, and the wavelengths withinthe background spectral band can correspond to wavelengths within arange from λ_(c)−Δλ/2 to λ_(c)+Δλ/2.

N can be greater than 3 (e.g., greater than 5). The sample can include Mnon-endogenous spectral contributors, where M≤N. M can be greater than 4(e.g., greater than 6). The methods can include displaying theautofluorescence image on a display device.

The methods can include determining, at each of multiple locations inthe sample, an amount of autofluorescence emission from the sample. Themethods can include, at each of the multiple locations in the sample,and for one or more of the N sample images, adjusting valuescorresponding to sample emission intensity to correct forautofluorescence emission from the sample based on the amount ofautofluorescence emission at each of the multiple locations and at leastone pure spectrum of autofluorescence emission from the sample. The atleast one pure spectrum of autofluorescence emission can includemultiple pure spectra of autofluorescence emission, and the multiplepure spectra of autofluorescence emission can each correspond to adifferent subset of the multiple locations. The methods can includedecomposing at least some of the N sample images based on the amount ofautofluorescence emission from the sample at each of the multiplelocations to obtain M spectral contributor images, where each of the Mspectral contributor images corresponds to light emission only from adifferent one of the non-endogenous spectral contributors, and at eachof the multiple locations, determining an amount of the M non-endogenousspectral contributors in the sample.

The methods can include decomposing the at least some of the N sampleimages based on at least one pure spectrum of autofluorescence emissionfrom the sample. The at least one pure spectrum of autofluorescenceemission can include multiple pure spectra of autofluorescence emission,and the multiple pure spectra of autofluorescence emission can eachcorrespond to a different subset of the multiple locations.

A sum of spectral emission intensities of each non-endogenous spectralcontributor in the sample at each wavelength within the backgroundspectral band can be 10% or less of a total fluorescence emissionintensity in the background spectral band.

The methods can include classifying pixels of one or more of the sampleimages into different classes based on information derived from theautofluorescence image. The different classes can correspond todifferent cell types in the sample.

The M non-endogenous spectral contributors can include one or morefluorescent species that selectively bind to different chemical moietiesin the sample. The one or more fluorescent species can include one ormore immunofluorescent probes. The M non-endogenous spectralcontributors can include one or more counterstains.

Embodiments of the methods can also include any of the other featuresdescribed herein, including any combinations of features individuallydescribed in connection with different embodiments, unless expresslystated otherwise.

In another aspect, the disclosure features computer readable storagemedia that include instructions that, when executed by a processingdevice, cause the processing device to: decompose a plurality of sampleimages of a biological sample using autofluorescence information for thesample to obtain one or more non-endogenous spectral contributor imagesof the sample; determine an amount of one or more non-endogenousspectral contributors at multiple locations in the sample based on theone or more non-endogenous spectral contributor images; and generate anoutput display that includes at least one of the non-endogenous spectralcontributor images on a display device connected to the processingdevice, where each sample image corresponds to a different combinationof a wavelength band of illumination light used to illuminate the sampleand one or more wavelength bands of emission light from the sample, theone or more wavelength bands of emission light defining a wavelengthrange, where the background image corresponds to illumination of thesample with light in a background excitation band and measurement oflight emission from the sample in a background spectral band, and wherefor each of the one or more non-endogenous spectral contributors in thesample illuminated with light in the background excitation band, aspectral emission intensity at each wavelength within the backgroundspectral band is 10% or less of a maximum measured spectral emissionintensity of the non-endogenous spectral contributor followingexcitation of the sample in each wavelength band of the illuminationlight and the background excitation band.

Embodiments of the storage media can include any one or more of thefollowing features. The storage media can include instructions that,when executed by the processing device, cause the processing device toobtain the autofluorescence information for the sample from a backgroundimage of the sample. Each non-endogenous spectral contributor image cancorrespond only to light emission from a different one of thenon-endogenous spectral contributors in the sample.

For each of the one or more non-endogenous spectral contributors in thesample illuminated with light in the background excitation band, thespectral emission intensity at each wavelength within the backgroundspectral band can be 4% or less (e.g., 2% or less) of the maximummeasured spectral emission intensity of the non-endogenous spectralcontributor following excitation of the sample in each wavelength bandof the illumination light and the background excitation band.

The sample can include M non-endogenous spectral contributors, whereM≤N. M can be greater than 4 (e.g., greater than 6).

The autofluorescence information can include an amount ofautofluorescence emission from the sample at each of the multiplelocations in the sample. The storage media can include instructionsthat, when executed by the processing device, cause the processingdevice to decompose the plurality of sample images using theautofluorescence information and at least one pure spectrum ofautofluorescence emission from the sample. The storage media can includeinstructions that, when executed by the processing device, cause theprocessing device to determine the at least one pure spectrum ofautofluorescence from the background image.

The at least one pure spectrum of autofluorescence emission from thesample can include two or more different pure spectra ofautofluorescence emission from the sample, and each of the differentpure spectra can correspond to a different subset of the multiplespatial locations.

The storage media can include instructions that, when executed by theprocessing device, cause the processing device to adjust valuescorresponding to sample emission intensity in each of the sample imagesto correct for autofluorescence emission from the sample based on theamount of autofluorescence emission at each of the multiple locationsand the at least one pure spectrum of autofluorescence emission from thesample.

A sum of spectral emission intensities of each non-endogenous spectralcontributor in the sample at each wavelength within the backgroundspectral band can be 10% or less of a total fluorescence emissionintensity in the background spectral band.

The storage media can include instructions that, when executed by theprocessing device, cause the processing device to classify pixels of oneor more of the sample images into different classes based on theautofluorescence information. The different classes can correspond todifferent cell types in the sample.

The M non-endogenous spectral contributors can include one or morefluorescent species that selectively bind to different chemical moietiesin the sample. The M non-endogenous spectral contributors can include ormore counterstains.

Embodiments of the storage media can also include any of the otherfeatures described herein, including any combinations of featuresdescribed in connection with different embodiments, unless expresslystated otherwise.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this disclosure belongs. Although methods and materialssimilar or equivalent to those described herein can be used in thepractice or testing of the subject matter herein, suitable methods andmaterials are described below. All publications, patent applications,patents, and other references mentioned herein are incorporated byreference in their entirety. In case of conflict, the presentspecification, including definitions, will control. In addition, thematerials, methods, and examples are illustrative only and not intendedto be limiting.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description. Other features and advantages will beapparent from the description, drawings, and claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart showing example steps for measuringautofluorescence of a sample.

FIG. 2 is a schematic diagram of an example system for measuring sampleautofluorescence.

FIG. 3 is a schematic diagram showing an example set of spectralemission bands that can be used to measure fluorescence emission due tonon-endogenous spectral contributors in a sample, and sampleautofluorescence.

FIG. 4 is a schematic diagram showing fluorescence emission from anon-endogenous spectral contributor in a sample.

FIGS. 5A-5D are graphs showing spectral properties of four epi-filtercubes that can be used to measure sample fluorescence.

FIG. 6A is a graph showing dye excitation responses for a counterstainand six dyes used as immuno-fluorescent labels.

FIG. 6B is a graph showing fluorescence emission responses for thecounterstain and dyes of FIG. 6A.

FIG. 6C is a graph showing autofluorescence emission from aformalin-fixed, paraffin-embedded lung cancer sample when excited at 387nm, and when excited at 425 nm as 604.

FIGS. 7A-7H are sample images and a background image obtained followingexcitation of a sample and measurement of sample fluorescence indifferent combinations of spectral excitation wavelength bands andspectral emission wavelength bands, each of the images showing a portionof the sample.

FIGS. 8A-8H are pure spectral contributor images and an autofluorescenceimage for the sample of FIGS. 7A-7H, obtained from unmixing the sampleand background images, each of the images showing a portion of thesample.

FIGS. 9A-9H are pure spectral contributor images and an autofluorescenceimage for the sample of FIGS. 7A-7H, obtained from unmixing the sampleand background images, each of the images showing the entire sample.

FIGS. 10A and 10B are sets of images, each set including anautofluorescence image, a binary classification mask, a FoxP3 abundanceimage, a threshold segmentation mask, and an image showing onlyFoxP3-positive cells, for a sample prepared with multiple non-endogenousdyes and for an unstained sample, respectively.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Various techniques have been developed in order to accurately measureimmuno-fluorescence targets in the presence of autofluorescence. Oneapproach involves multi spectral imaging, where sample images areobtained at different wavelengths to distinguish the contributions ofdyes associated with immuno-fluorescent labeling from fluorescenceemission due to sample autofluorescence. While these signals overlapspectrally, the spectral shape of each signal is different. A spectralunmixing or other spectral decomposition step is performed to separatecontributions to the measured fluorescence signals into individualspectral components corresponding to the individual dyes, enablingmeasurement of multiple dyes despite the presence of spectrally broadautofluorescence emission. Spectral unmixing techniques can distinguishcontributions from a number of dyes in a sample, even when thefluorescence emission from the dyes overlaps spectrally, providing a keymultiplex sample labeling advantage.

Multiplexed immuno-fluorescence labeling refers to targeting multipleprimary antibodies in one sample, and it can be achieved in variousways. Conceptually the simplest is direct labeling, where a primaryantibody molecule is conjugated to a dye, with different dyes conjugatedto different primary antibodies. This labeling process typically resultsin one dye molecule per target, so signal levels can be low unless thetarget itself is abundant. Indirect labeling provides a way to couplemultiple dye molecules per target, enabling weaker targets to bestudied. To avoid cross-labeling, different secondary antibody isotypesor species are generally used for each target, making the labelingprocess more complex.

Another approach to introducing multiple specific dyes is to conjugate aprimary antibody to a nucleotide sequence that is recognized by acomplementary sequence conjugated to a dye polymer. This technique isnot limited by the number of available secondary antibody species.

To increase the amount of each fluorescent dye that selectively binds tothe sample, dyes can be selectively applied to the sample sequentiallyusing tyramide signal amplification (TSA). In TSA, a single primaryantibody is used together with a secondary antibody coupled tohorseradish peroxidase to catalyze covalent binding of dye molecules tosurrounding tissue through cinnamide or tyramide molecules. Because thedye molecules are covalently bound to the tissue, the primary andsecondary antibodies can subsequently be removed and the process can berepeated with other primary antibodies and dye molecules, permittinghigher concentrations of dye molecules to be introduced into the samplein a multiplexed, specific manner.

In some TSA-based staining protocols, further amplification can beachieved by covalently binding a small molecule ligand selectively tothe sample tissue via cinnamide or tyramide molecules, followed byincubation of the sample with a ligand-binding protein that isconjugated to multiple dyes molecules. For example, biotin can beintroduced as the ligand and streptavidin can be used as theligand-binding protein, but other ligand-protein combinations can alsobe used. Because the dye molecules are not covalently bound to thetissue, incubation of the ligand-bound sample with the ligand-bindingprotein typically occurs at the end of the multiplex staining protocol.

Methods and systems have been developed for the purpose of performingmultispectral imaging of whole slide samples. In particular, suchmethods and systems can generate mosaic images with diffraction-limitedmultispectral images with a multi-centimeter field of view in twodimensions across a sample. Such systems use multi-band filters andselectable light sources such as LEDs or lasers, multiple epi-filtersets, or combinations of the two. When multiple epi-filters are used,registration of the multispectral images is important, as it allowsimages corresponding to different portions of the sample to be combinedto form a seamless mosaic. US Patent Application Publication No. US2014/0193061 describes methods for using multiple filters, each of whichimages the sample at a common band such as a counterstain band (e.g., aDAPI counterstain band) as well as other bands, and for combining theimages into a mosaic image of the sample by using images measured at thecommon band to register images corresponding to all of the sampleimaging bands. In this manner, samples can be imaged in 10 or moredistinct emission bands can be imaged and co-registered. The entirecontents of US Patent Application Publication No. US 2014/0193061 areincorporated herein by reference.

The foregoing methods for performing multispectral sample imaging candetermine contributions from sample autofluorescence during spectralunmixing. For example, using an estimate of a pure spectrum of sampleautofluorescence, the spectral unmixing process can yield anautofluorescence abundance image that represents a spatial distributionof autofluorescence within the sample. The autofluorescence image can beviewed as a type of “remainder” image that accounts for samplefluorescence that is not attributable to dyes applied to the sample.

For certain samples, however, abundance measurements of autofluorescencethat are derived purely from spectral unmixing may not be entirelyaccurate. In particular, such measurements are derived using an estimateof a pure autofluorescence spectrum which may not perfectly correspondto autofluorescence in a particular sample of interest, particularlywhen that sample also include multiple fluorescent dyes and is preservedaccording to a certain fixation protocol. Moreover, autofluorescence canvary in a sample among different sample regions—and in particular, amongdifferent structures within a sample, such as different types of cells,and different types of cellular features (e.g., stroma, cytoplasm, cellmembranes, nuclei)—and this variation may not be adequately representedby a single estimate of a pure autofluorescence spectrum.

The methods and systems described herein allow autofluorescence in asample to be measured directly, rather than simply extracted during aspectral unmixing process. By directly measuring autofluorescence,sample-specific autofluorescence responses can be properly measured andaccounted for during analysis of multispectral images that represent oneor more dyes applied to the sample to target specific antibodies,proteins, nucleic acids, and other sample structures and cellularcomponents. In particular, measured fluorescence signals in spectralemission bands that correspond to the individual applied dyes can becorrected to account for sample autofluorescence prior to, or during,image analysis, providing for more accurate abundance measurements ofeach of the individual applied dyes. To permit these improvements, boththe abundance and spatial distribution of sample autofluorescence atmultiple locations within the sample is determined prior to spectraldecomposition of the multispectral images corresponding to theindividual applied dyes.

Sample imaging systems typically include one or more displays on whichsample images that represent spectral contributions corresponding toindividual applied dyes are displayed to pathologists or otherphysicians or technicians for purposes of diagnosing disease and otherstates within the samples. Dyes are typically selected to act asreporters for particular antibodies, proteins (e.g., trans-membraneproteins), nucleic acid fragments, and/or cellular structures ofinterest which may indicate the presence or absence of certainconditions, and may provide complex physiological information about cellmigration, protein expression, regulatory networks, mutations, and otherevents in the sample. The methods and systems described herein permitmore accurate spectral contributor images to be displayed, therebysignificantly improving conventional sample imaging and display systems,which in turn improves diagnostic information rendered by such systems.

The autofluorescence image(s) that are measured according to the methodsand systems described herein can be also be used for additionalpurposes. For example, a measured autofluorescence image can be analyzedusing a trained pixel-based classifier to identify different types ofcells or cellular structures in a sample, and to exclude regions of asample that correspond to certain types of cells or cellular structuresthat are not of interest. For example, a trained classifier can be usedto identify red blood cells in a sample, which may not be of interest,but which may emit fluorescence in spectral regions that correspond toone or more dyes that have been applied to the sample to target specificsample targets of interest. A trained classifier can also be used toidentify other types of cells, including (but not limited to) differenttypes of white blood cells such as lymphocytes, neutrophils,eosinophils, monocytes, and basophils; reticulocytes; and tumor cells.

Regions of the sample that are not of interest can be excluded from thesubsequent analysis (e.g., spectral unmixing of sample images), therebyexcluding regions from the sample that would otherwise represent falsepositive detection for one or more of the applied dyes, and therefore,the cellular component(s) targeted by those applied dyes. By excludingcertain sample regions from analysis via classification operationsperformed on measured autofluorescence images, the methods and systemsdescribed herein permit more accurate spectral contributor images to bedisplayed, thereby further significantly improving conventional sampleimaging and display systems, and in turn improving the quality andutility of diagnostic information rendered by such systems.

The methods and systems described herein enable relatively rapid,whole-slide, multispectral imaging of multiply stained samples, foraccurate and quantitative analysis in applications that includeimmuno-oncology assays, cell signaling studies, and multiplexedimmuno-fluorescence pathology experiments generally. The methods andsystems can be applied to a wide variety of samples, includingbiological samples such as (but not limited to) formalin-fixed,paraffin-embedded (FFPE) samples that have been labeled using a varietyof different methods.

FIG. 1 is a flow chart 100 showing a series of example steps for imaginga sample. In a first step 102, a sample is prepared for imaging byapplying one or more non-endogenous dyes to the sample. As used herein,a “dye” is a non-endogenous substance that binds to structures/chemicalmoieties within a sample, and emits fluorescent light when exposed toillumination light. The term “dye” is used interchangeably with the term“stain”; for purposes of this disclosure, a “dye” and a “stain”correspond to the same substances.

In general, one or more dyes are applied to the sample in step 102. Forexample, the number of applied dyes can be two or more (e.g., three ormore, four or more, five or more, six or more, seven or more, eight ormore, nine or more, ten or more, 12 or more, 15 or more, 20 or more, oreven more). Applied dyes can be bound to specific types of antibodies,proteins, nucleic acids, vesicles, lipids, or other substances withinthe sample. Applied dyes can also be bound to specific cell types (e.g.,red blood cells, lymphocytes, T cells, B cells) within a sample.Further, applied dyes can be bound to specific cellular structures orcompartments (e.g., stroma, cell membranes, cytoplasm, cell nuclei,mitochondria, golgi bodies) within a sample.

In some embodiments, the one or more dyes applied to the sample caninclude one or more counterstains that bind to multiple structures,regions, or components in sample cells. Examples of suitablecounterstains include, but are not limited to, DAPI, DRAQ5, Hoechst33258, Hoechst 33342, and Hoechst 34580.

In certain embodiments, one or more of the dyes that are applied to thesample have spectrally separated emission bands. By choosing dyes withthis property, fluorescence emission from each dye can readily beisolated from emission due to the other dyes, and measured, usingsuitable emission filters, minimizing interference resulting fromspectral emission cross-talk among the dyes. As an example, in someembodiments, dyes are selected such that for a given pair of dyes D1 andD2, each having an emission spectrum and a maximum emission intensityD1_(max) and D2_(max) at wavelengths λ_(D1) and λ_(D2) respectivelywithin the dye's emission spectrum, the emission intensity of dye D1 atλ_(D2) is less than 10% (e.g., less than 8%, less than 6%, less than 4%,less than 2%, less than 1%, less than 0.5%, less than 0.25%) ofD1_(max), and the emission intensity of dye D2 at λ_(D1) is less than10% (e.g., less than 8%, less than 6%, less than 4%, less than 2%, lessthan 1%, less than 0.5%, less than 0.25%) of D2_(max). The dyes may beselected such that the above pairwise relationship holds among allmembers of a group of three or more dyes, four or more dyes, five ormore dyes, six or more dyes, eight or more dyes, ten or more dyes, 12 ormore dyes, 15 or more dyes, 20 or more dyes, or even more dyes.

As discussed above, a wide variety of different sample preparationprotocols can be used in step 102. One example of a suitable preparationprotocol is described below. However, it should be understood that manydifferent protocols that apply many different dyes to samples can beused.

Samples can be prepared for multispectral imaging using the Opal®Multiplex immunohistochemical (IHC) reagents (available from AkoyaBiosciences, Menlo Park, Calif.), which can be used to label a widevariety of molecular targets within a sample. FFPE tissue samples areprepared for staining with Opal® reagents by baking at 60° C. for onehour followed by three 10-minute washes in xylene to remove paraffin.Samples are then rehydrated via an ethanol gradient into deionized waterand fixed using 10% neutral buffered formalin for 20 minutes, followedby a wash in deionized water.

The Opal® Multiplex IHC staining procedure begins with one round ofantigen retrieval, which can be performed using either buffer AR6(antigen retrieval pH 6) or AR9 (antigen retrieval pH 9) via microwavetreatment. After antigen retrieval, each sample target is labeledsequentially in a cycle that includes five steps: blocking, primaryantibody incubation, secondary antibody incubation, tyramide deposition,and antibody stripping.

Blocking is achieved by incubating the sample for 10 minutes in Opal®Antibody Diluent at room temperature. The primary antibody can beincubated at various times and temperatures depending on the target. Forexample, for a lung cancer sample, Table 1 lists an example of primaryantibodies and dilution ratios for a variety of different sampletargets.

TABLE 1 Primary Primary Antibody Antibody Tyramide Tyramide StainingVendor, Catalog Dilution Reagent and Dilution Order Target No., Clone,Species Factor Vendor Factor 1 FoxP3 Abeam 1:100 Opal 570 (Akoya 1:300(Cambridge, UK), Biosciences, ab20034, 236A/E7, Menlo Park, CA) a-Ms 2PD-L1 CST (Danvers 1:300 Opal 520 (Akoya 1:150 MA), 13684, Biosciences,E1L3N, a-Rb Menlo Park, CA) 3 PD-1 AbCam 1:300 Opal 690 (Akoya 1:100(Cambridge, UK), Biosciences, ab137132, Menlo Park, CA) EPR4877(2), a-Rb4 CD68 Dako (Santa Clara 1:100 Opal 620 (Akoya 1:150 CA), M0876, PG-Biosciences, M1, a-Ms Menlo Park, CA) 5 CD8 AbD Serotec 1:300 Dy4301:450 (Oxford UK), (Dyomics, Jena MCA1817, 4B11, Germany) a-Ms 6 Pan-Novus (Littleton 1:300 TSA-biotin 1:50  Cytokeratin CO), NBP2-29429,(Akoya AE1/AE2, a-Ms Biosciences, Menlo Park, CA)

Each of the antibodies can be incubated for approximately 30 minutes atroom temperature. Following primary antibody incubation, secondaryantibody incubation is performed using the Opal® Polymer HRP Ms+Rbsolution (Akoya Biosciences, Menlo Park, Calif.) for 10 minutes at roomtemperature, followed by 3 rinses with a TBST buffer composed of 0.1 MTRIS-HCl, pH 7.5, 0.15 M NaCl and 0.05% Tween® 20 (Sigma-Aldrich, St.Louis Mo.). Tyramide deposition is performed at room temperature for 10minutes followed by 3 rinses with TBST. A different tyramide reagent ispaired with each primary antibody, according to the targets chosen anddyes selected to interrogate the chosen targets. A staining cycle iscompleted by antibody stripping via microwave treatment in AR6 or AR9.The foregoing 5-step cycle can be repeated for each target in thesample. In direct analogy with the above procedure, if ligand-basedtyramide reagents are used in the sample preparation procedure,incubation with the dye-conjugated ligand-binding proteins occurs afterthe final antibody stripping step.

After each of the dyes corresponding to particular targets within thesample have been applied, one or more counterstains can optionally beapplied to the sample by incubating the sample with the counterstain.For example, to apply DAPI counterstain to a sample, the sample can beincubated in a DAPI solution (4 drops/mL) for 5 minutes at roomtemperature, followed by one wash in deionized water and one wash inTBST.

A stained sample—to which one or more dyes associated with specificsample targets and optionally, one or more counterstains, have beenapplied—can be further prepared for imaging by applying a mountingmedium (for example, ProLong® Diamond, available from ThermoFisherScientific, Waltham Mass.) and a coverslip.

In some embodiments, sample preparation can also include preparation ofunstained samples that can be used to perform autofluorescencemeasurements. In general, unstained samples can be prepared by followinga procedure that is analogous to the procedure for preparing stainedsamples, and simply omitting incubation steps that include fluorescentdyes. In the example described above, incubation steps with the variousdyes shown in the fifth column of Table 1 are omitted, but the steps areotherwise the same. Such preparation protocols yield an unstained samplethat has been processed with the same temperature changes, antibodyadditions, and pH changes that stained slides undergo, and to the extentthat these processing steps affect the endogenous autofluorescenceemission from the sample, the effect should be similar for both stainedand unstained samples.

In general, a wide variety of different sample preparation techniquesusing different combinations of dyes specific to certain sample targets,counterstains, and other sample labeling moieties can be used. Thedevelopment of a specific multiplexed immunofluorescent samplepreparation protocol for a particular purpose typically involvesselection of primary antibodies, improvement/optimization of antibodyand dye dilutions, and other immunohistochemical process developmentsteps to maintain sample integrity and enhance selective dye binding tothe sample. All such protocols can generally be used in the methods andsystems described subsequently, except where expressly stated otherwise.

Returning to FIG. 1, after sample preparation is complete in step 102,the prepared sample is imaged in step 104 to obtain one or more sampleimages. Each of the sample images obtained in step 104 corresponds to aselected combination of illumination light within an excitation band,and fluorescence emission from the sample in response to theillumination light, measured in one or more emission bands.

A variety of different sample imaging systems can be used to obtainsample images. An example of one sample imaging system 200, implementedas a fluorescence microscope, is shown in FIG. 2. System 200 includes alight source 202, a dichroic mirror 204, an optical filter 206(implemented as an excitation filter 206 a and an emission filter 206b), an objective lens 208, a stage 210, and a detector 212. Each ofthese components is coupled to a controller 214 that includes aprocessing device 216.

Controller 214 transmits and receives control and data signals from eachof the system components, and can therefore control each of thecomponents (more specifically, processing device 216 controls each ofthe system components). In certain embodiments, all of the steps and/orcontrol functions described in connection with system 200 are performedby controller 214. Alternatively, in some embodiments, certain steps canbe performed by a human operator of system 200.

Light source 202 is an adjustable source that can produce light having avariable distribution of illumination wavelengths. In some embodiments,for example, light source 202 includes a plurality of LEDs of differentwavelengths that can be selectively activated by controller 214 togenerate illumination light having desired spectral properties. Incertain embodiments, light source 202 includes one or more laser diodes,lasers, incandescent sources, and/or fluorescent sources, each of whichis controllable by controller 214.

The illumination light generated by light source 202 reflects fromdichroic mirror 204 and is incident on optical filter 206. Filter 206typically includes multiple filters, each of which can be selectivelyinserted into the path of the illumination light. Each of the filtershas an associated excitation spectral band and one or emission spectralbands. Controller 214 adjusts filter 206 based on the illumination lightgenerated by light source 202 to allow light of a suitable wavelengthdistribution to be incident on the sample.

Filter 206 can be implemented in various ways. In some embodiments, forexample, filter 206 includes a plurality of different epi-filter cubes,each of which can be selectively rotated into the path of theillumination light by controller 214. In certain embodiments, filter 206includes an adjustable filtering element (e.g., a liquid crystal basedelement) for which excitation and emission spectral bands can beselectively chosen by controller 214. Other implementations of filter206 can also be used in system 200.

The filtered illumination light emerging from filter 206 is then focusedby lens 208 onto the surface of sample 250, which is supported by aslide 252 that is mounted on a stage 210. Stage 210 permits movement ofsample 250 in each of the x- and y-directions, and is controllable bycontroller 214. Motion of stage 210 in the x- and y-directions allowsthe filtered illumination light to be directed to different regions ofthe sample. By moving the sample relative to the focal region of theillumination light, illumination light can be directed to multipledifferent regions of the sample, permitting whole-slide imaging ofsample 250.

The filtered illumination light generates fluorescence emission fromsample 250, and the fluorescence that is emitted in the direction oflens 208 is collimated by lens 208 and passes through filter 206. Asdiscussed above, filter 206—which can be adjusted by controller214—defines one or more emission spectral bands. The fluorescenceemission from sample 250 is filtered by filter 206 such that only lightwithin the one or more emission spectral bands is transmitted by filter206. The filtered fluorescence emission light is transmitted by dichroicmirror 204 and detected by detector 212.

Detector 212 can be implemented in various ways. In some embodiments,for example, detector 212 includes a CCD-based detection element. Incertain embodiments, detector 212 includes a CMOS-based detectionelement. Detector 212 can also optionally include spectrally-selectiveoptical elements such as one or more prisms, gratings, diffractiveelements, and/or filters, to permit wavelength-selective detection ofthe filtered fluorescence emission light. In response to the incidentfiltered fluorescence emission light, detector 212 generates one or moreelectronic signals that represent quantitative measurements of thefiltered fluorescence emission light. The signals are transmitted tocontroller 214 which processes the signals to extract measurementinformation corresponding to sample 250.

During operation, system 200 typically captures N different sampleimages, each of which corresponds to a different combination of anexcitation wavelength band of filter 206 (which controls the spectraldistribution of illumination light that is incident on sample 250) andone or more emission wavelength bands of filter 206 (which control thespectral distribution of fluorescence emission from sample 250 that isdetected by detector 212). N is 1 or more, and in general can be anynumber (e.g., 2 or more, 3 or more, 4 or more, 5 or more, 6 or more, 7or more, 8 or more, 9 or more, 10 or more, 12 or more, 15 or more, 20 ormore, or even more).

In certain embodiments, N can be related to the number of non-endogenousspectral contributors in the sample. As used herein, a “non-endogenousspectral contributor” is a component of sample 250 that has been addedto the sample, and that emits fluorescence when excited by illuminationlight from light source 202. Non-endogenous spectral contributorsinclude the one or more dyes applied to sample 250, as discussed above.As an example, N can be equal to or greater than M, the number ofnon-endogenous spectral contributors in the sample.

Returning to FIG. 1, system 200 also typically captures one or morebackground images of the sample in step 106. The background image(s) canbe captured after, before, or interleaved with, the capturing of thespectral images of the sample in step 104. The background image(s)correspond(s) to a combination of an excitation wavelength band offilter 206 and a background spectral band of filter 206. An importantaspect of the background spectral band is that it is selected such thatthe non-endogenous spectral contributors in sample 250 do not generatesignificant fluorescence emission light within the background spectralband.

One example of a set of fluorescence emission spectral wavelength bandsand the background spectral wavelength band is shown schematically inFIG. 3. In FIG. 3, seven different fluorescence emission spectralwavelength bands, B1-B7, are shown, along with a background spectralwavelength band, X. Each of the bands corresponds to a respectivewavelength λ₁-λ₇ and λ_(X), respectively, which represents the centralwavelength of the spectral band, as determined from the full-width athalf maximum (FWHM) spectral shape of each band. It should be noted thatthe discussion below refers to seven emission spectral wavelength bandsB1-B7 for illustrative purposes only. In general, any number N ofspectral wavelength bands can be used when capturing sample images, asdiscussed above.

The wavelengths λ₁-λ₇ define a wavelength range of the fluorescenceemission spectral wavelength bands B1-B7. In some embodiments, thebackground spectral band X is selected such that the wavelength λ_(X)falls within the wavelength range of the fluorescence emission spectralwavelength bands B1-B7. Alternatively, in certain embodiments, λ_(X)falls outside this range. It can be advantageous for λ_(X) to be withinthe wavelength range of the fluorescence emission spectral wavelengthbands B1-B7 in certain implementations, as selecting the backgroundspectral wavelength band in this manner allows for a larger spectralrange to be dedicated to the fluorescence emission spectral wavelengthbands, making isolation of the various fluorescence emission signalsfrom sample 250 easier.

It should be noted that while in FIG. 3 the background spectral band Xis separated spectrally from spectral wavelength bands B1-B7, this isnot always the situation. In some embodiments, background spectral bandX overlaps with portions of one or more spectral wavelength bands B1-B7.In general, background spectral band X can be selected such that itoverlaps spectrally with none of the spectral wavelength bandscorresponding to the non-endogenous spectral contributors, oralternatively, with one or more (e.g., two or more, three or more, fouror more, or even more) of the spectral wavelength bands corresponding tothe non-endogenous spectral contributors. As described below, even whenspectral overlap occurs between background spectral band X and one ormore of the spectral wavelength bands B1-B7, endogenous sampleautofluorescence can still be measured by exciting the sample in aselected wavelength band, and then measuring sample autofluorescence inbackground spectral band X.

In general, the background image of the sample is obtained by detectingfluorescence emission from the sample in background spectral band X. Incertain embodiments, some or all background images of the sample areobtained by detecting fluorescence emission from the sample in more thanone background spectral band. Each of the background spectral bands thatare used to detect fluorescence emission from the sample generallyshares a common attribute: each of the non-endogenous spectralcontributors in the sample does not generate significant fluorescenceemission in the background spectral band when the sample is excited in acertain excitation wavelength band. Accordingly, by exciting the samplein a selected excitation wavelength band and measuring samplefluorescence in the background spectral band, a background image of thesample can be obtained which includes spectral contributions fromsubstantially only sample autofluorescence, without significantcontributions from non-endogenous spectral contributors such as applieddyes and counterstains.

FIG. 4 is a schematic diagram showing an example of the nature of therelationship between fluorescence emission from the non-endogenousspectral contributors and the background spectral band. To image asample, the sample is exposed to illumination light in a band W_(exc,i),and fluorescence emission from the excited sample is measured in anemission band W_(emi,j). This procedure is repeated to obtain N sampleimages, each corresponding to a different combination of excitation bandW_(exc,i) and emission band W_(emi,j). To obtain a background image ofthe sample, the example is exposed to illumination light in a backgroundexcitation band W_(back,exc) and sample fluorescence is measured in abackground spectral band W_(back,emi).

For each non-endogenous spectral contributor in the sample, the spectralcontributor will exhibit a maximum fluorescence intensity I_(max) at aparticular wavelength within the wavelength range defined by theemission bands W_(emi,j). The maximum fluorescence intensity I_(max)results from excitation with light in one of the excitation bandsW_(exc,i) and W_(back,exc). In the methods and systems described herein,the background excitation band W_(back,exc) and the background spectralband W_(back,emi) are selected such that for any wavelength in thebackground spectral band W_(back,emi), the intensity of the fluorescenceemission from each non-endogenous spectral contributor is 10% or less(e.g., 8% or less, 6% or less, 5% or less, 4% or less, 3% or less, 2% orless, 1% or less, 0.5% or less, 0.25% or less, 0.1% or less, 0.05% orless, 0.01% or less, or even less) than the maximum fluorescenceintensity I_(max) of that non-endogenous spectral contributor.

In FIG. 4, a fluorescence emission intensity spectrum E(λ) for anon-endogenous spectral contributor excited by illumination light in thebackground excitation band W_(back,exc) is shown. Also shown is thebackground spectral band W_(back,emi) (which corresponds to spectralband X). Background spectral band W_(back,emi) corresponds to awavelength λ_(X) (as discussed above) and includes a band of wavelengthsλ_(X1)-λ_(X2). To avoid fluorescence emission cross-talk in thebackground spectral band, for each wavelength in the rangeλ_(X1)-λ_(X2), the fluorescence emission intensity E(λ) of thenon-endogenous spectral contributor is 10% or less (e.g., 8% or less, 6%or less, 5% or less, 4% or less, 3% or less, 2% or less, 1% or less,0.5% or less, 0.25% or less, 0.1% or less, 0.05% or less, 0.01% or less,or even less) of the maximum fluorescence emission intensity I_(max) forthe non-endogenous spectral contributor excited in any of the wavelengthbands W_(exc,i) and W_(back,exc). By selecting the background spectralband X in such a manner, the background image of the sample that iscaptured by the system corresponds nearly entirely (or entirely) toautofluorescence emission from the sample.

In general, the background excitation and background spectral bands canbe selected such that the above relationship holds for some or allnon-endogenous spectral contributors in the sample. Moreover, when thebackground image of the sample is captured by detecting fluorescenceemission in multiple background spectral bands, then the multiplebackground spectral bands can be selected such that the foregoingrelationship holds for some or all of the background spectral bands (andsome or all of the non-endogenous spectral contributors).

The background spectral band in FIG. 4 has a spectral shape thatcorresponds approximately to a square or “top hat” distribution. Theedges of the distribution therefore define the wavelength rangeλ_(X1)-λ_(X2) associated with the background spectral band. Whenbackground spectral band has a more complex shape, then the wavelengthrange associated with the background spectral band is determined basedon the FWHM spectral range of background spectral band. Specifically,for a background spectral band having a FWHM spectral range of Δλ,centered at a wavelength λ_(c), then the wavelength range associatedwith background spectral band extends from λ_(X1)=λ_(c)−Δλ/2 toλ_(X)2=λ_(c)+Δλ/2. The background spectral band can be selected suchthat the relationship described above between the fluorescence emissionfrom the non-endogenous spectral contributors in the sample and thebackground spectral band holds within the wavelength range λ_(X1) toλ_(X2).

The background excitation band and background spectral band can also beselected according to other criteria to ensure that fluorescenceemission cross-talk into the background spectral band remains relativelylow. In some embodiments, for example, the background excitation bandand background spectral band can be chosen such that for some or allnon-endogenous spectral contributors, the integrated intensity of thenon-endogenous spectral contributor in the background spectral band(i.e., the emission intensity summed between wavelengths λ_(X1) andλ_(X2)) after excitation in the background excitation band is less than5% (e.g., less than 4%, less than 3%, less than 2%, less than 1%, lessthan 0.5%, less than 0.3%, less than 0.2%, less than 0.1%, less than0.05%, less than 0.01%) of the maximum integrated fluorescence emissionintensity for the non-endogenous spectral contributor across allwavelengths in the wavelength range defined by the emission bandsW_(emi,j) when the non-endogenous spectral contributor is excited in theexcitation bands W_(exc,i).

In some embodiments, the background excitation band and backgroundspectral band can be chosen such that when the sample is illuminatedwith light in the background excitation band, the integrated intensityof fluorescence emission from all non-endogenous spectral contributorsin the sample is 10% or less (e.g., 8% or less, 6% or less, 4% or less,2% or less, 1% or less, 0.5% or less, 0.25% or less, 0.1% or less, 0.05%or less, or even less) than the integrated intensity of all measuredfluorescence emission in the background spectral band (i.e., due to bothnon-endogenous spectral contributors and sample autofluorescence).

It should also be recognized that in some embodiments, the backgroundexcitation band and background spectral band can be selected such thatmore than one of the foregoing conditions is satisfied.

Returning to FIG. 1, in the next step 108, an autofluorescence image ofthe sample is obtained. In some embodiments, due to the selection of thebackground spectral band X, the background image of the sampleeffectively corresponds substantially only to autofluorescence from thesample. Accordingly, the sample autofluorescence image correspondsdirectly to the sample background image, without any further processing.

In certain embodiments, the sample autofluorescence image can beextracted from the background image by decomposing the background imageusing a technique such as spectral unmixing. In spectral unmixing, thebackground image (which can be a multispectral image that includesemission intensity measurements at each of the wavelengths within thebackground spectral band X, and at each sample location of interest, toform a multispectral image cube) can be unmixed using pure estimates ofthe sample autofluorescence spectrum and the fluorescence emissionspectra of each of the non-endogenous spectral contributors in thesample, to obtain an autofluorescence image of the sample. Because thebackground image of the sample contains only small contributions fromeach of the non-endogenous spectral contributors in the sample, theunmixing process is typically highly effective at isolating sampleautofluorescence and generating an autofluorescence image. Suitablemethods for spectral unmixing are described, for example, in U.S. Pat.No. 7,321,791 and in PCT Patent Application Publication No. WO2005/040769, the entire contents of each of which are incorporated byreference herein.

Returning to FIG. 1, the sample images obtained in step 104 can bedecomposed in step 110 to obtain a set of pure spectral contributorimages, each of which includes substantially only contributions from oneof the non-endogenous spectral contributors in the sample. It should benoted that in FIG. 1, steps 108 can occur sequentially (i.e., theautofluorescence image can be obtained first, followed by the purespectral contributor images), or alternatively, steps 108 and 110 canoccur concurrently (i.e., both the autofluorescence image and the purespectral contributor images can be obtained at the same time, such asvia a single spectral umixing procedure).

By performing decomposition step 110, the distribution of each of theapplied dyes within the sample can be determined, and therefore, thedistribution of the molecular targets associated with each of the dyes.Spectral unmixing can be used to perform the decomposition in step 110.In certain embodiments, when the sample autofluorescence distribution isalready known from step 108, the spectral unmixing process used in step110 can take account of the autofluorescence abundance information. Atwo-step spectral unmixing procedure which uses previously determinedautofluorescence abundance information is discussed below.Alternatively, steps 108 and 110 can be performed concurrently in aone-step spectral unmixing procedure.

Next, in step 112, amounts of one or more of the non-endogenous spectralcontributors in the sample are determined. In effect, this stepcorresponds to determining quantitatively the distribution one or morenon-endogenous spectral contributors, and therefore, the quantitativedistributions of one or more molecular targets within the sample.Amounts of the non-endogenous spectral contributors can be determineddirectly using spectral unmixing, as the entries in the abundancematrix. Accordingly, in certain embodiments, steps 110 and 112 areperformed together, as spectral unmixing yields both the spatialdistribution and quantitative amounts of the non-endogenous spectralcontributors at some or all locations within the sample.

In step 114, the autofluorescence image obtained in step 108 canoptionally be displayed on a display interface or device 218 connectedto controller 214. Moreover, in step 114, any of the pure spectralcontributor images obtained in step 110 can optionally be displayed oninterface 218. As described above, a system that displays some of all ofthese images represents a notable improvement to conventionalfluorescence microscopes and other multispectral sample imaging devices,which are not able to determine sample autofluorescence and obtain purespectral contributor images in the same manner. In particular, systemsthat determine sample autofluorescence and obtain pure spectralcontributor images as discussed herein are able to acquire thisinformation more quickly than conventional sample imaging devices, withreduced data storage requirements relative to conventional devices. Forexample, the number of spectral bands and/or wavelengths in which samplefluorescence is measured to obtain an accurate quantitativedetermination of endogenous autofluorescence is significantly reducedrelative to conventional sample imaging devices. As a result, the amountof measured spectral information from the sample is reduced, andcomputationally intensive operations such as spectral unmixing proceedmore rapidly.

The procedure in flow chart 100 then terminates at step 116.

It should be noted that the steps shown in flow chart 100 are not theonly steps that can be performed in a sample analytical workflow. Othersteps can also be performed prior to, concurrent with, or after any ofthe steps shown in FIG. 1. For example, between steps 110 and 116, theprocedure shown in FIG. 1 can include an additional analysis step inwhich some or all of the pure spectral contributor images arequantitatively analyzed to determine one or more properties of thesample and/or its molecular targets. In general, any analyticalprocedures can be performed, including for example quantitativelydetermining statistical attributes associated with the distributions ofvarious molecular targets.

In general, the procedure described in FIG. 1 permits sampleautofluorescence to be quantitatively determined. Autofluorescence canbe determined separately from the determination of the quantitativedistributions of each of the non-endogenous spectral contributors in thesample. Further, in some embodiments, autofluorescence can be determinedprior to the determination of the quantitative distributions of each ofthe non-endogenous spectral contributors in the sample. Alternatively,in certain embodiments, autofluorescence and the quantitativedistributions of each of the non-endogenous spectral contributors aredetermined concurrently.

As a result, in some embodiments, the sample autofluorescence can beused to correct the raw spectral fluorescence emission measurements ofeach of non-endogenous spectral contributors. In certain embodiments, asdiscussed above, this correction occurs during decomposition (e.g.,spectral unmixing), where the sample autofluorescence distribution isused in a two-stage unmixing procedure.

Alternatively, in some embodiments, sample autofluorescence can be usedprior to unmixing of the sample images to adjust measured intensityvalues in the sample images directly. For example, in some or all of theraw multispectral sample images (and optionally, at some or all of thespatial locations within the images), measured spectral emissionintensity values can be adjusted based on the amount of autofluorescenceemission at corresponding locations in the sample, and at least one purespectrum of the sample autofluorescence emission. Adjustments can beperformed, for example, by subtracting autofluorescence contributionsfrom the measured spectral emission intensity values. In samples wheremore than one sample autofluorescence emission spectrum is present(e.g., due to differences in autofluorescence emission in differentsample regions, as will be discussed below), different pure spectra ofsample autofluorescence emission can be used to perform the correctionsin corresponding sample regions.

In general, the methods and systems described herein can use a widevariety of different filters to select suitable spectral excitation andemission bands for obtaining pure contributor and autofluorescenceimages of a sample. Further, in some embodiments, a sample backgroundimage can be obtained based on fluorescence emission measurements inmore than one background spectral band. For example, a sample backgroundimage can be obtained from fluorescence emission measurements in two ormore (e.g., three or more, four or more, five or more) backgroundspectral bands. In general, when a sample background image is obtainedfrom fluorescence emission measurements in more than one backgroundspectral band, some or all of the background spectral bands satisfy thecriteria discussed above.

For spectral unmixing operations, pure spectral estimates for the dyesand sample autofluorescence can be obtained in various ways. Forexample, dye spectra can be modeled based on the measured properties ofthe dyes and filters, as will be discussed below, but they can also bemeasured directly. For example, U.S. Pat. No. 10,126,242 describesmethods for extracting the pure component spectrum from a singly-stainedartifact sample, despite the presence of autofluorescence in the sampleused for the measurement. The entire contents of U.S. Pat. No.10,126,242 are incorporated herein by reference.

In certain embodiments, classification of pixel types can be combinedwith adaptive unmixing to improve unmixing accuracy. In other words,pixels in the autofluorescence image can be classified according to thetype of sample material at each pixel (e.g., stroma, extracellularmatrix, red blood cell, collagen), followed by unmixing at each pixelwith type-specific pure spectral estimates to improve unmixing accuracy.Alternatively, or in addition, certain pixels in sample images can beexcluded from further analysis based on the type of sample material towhich they correspond (i.e., to the classification type determined forthe pixel). For example, pixels classified as corresponding to red bloodcells, and/or collagen, and/or extracellular matrix may not be ofinterest. Those pixels can be designated as outside a region of interestfor the sample, and measured information corresponding to these pixels(e.g., spectral information) can subsequently be ignored. This can yielda significant reduction in analysis time for the sample. Informationmeasured for those pixels can also optionally be deleted, reducing datastorage requirements.

Conversely, in certain embodiments, pixels corresponding to samplestructures such as collagen, stroma, extracellular matrix, and red bloodcells can be preferentially analyzed and visualized by displaying anautofluorescence image in which pixels corresponding to samplestructures that are not of interest are excluded (i.e., displayed asdark pixels).

As part of the spectral unmixing, the autofluorescence spectrum of eachsample material type can be measured, resulting in type-specificautofluorescence spectra for the sample. These measurements can beperformed on an adjacent serial section of the same sample to providethe most representative data. Alternatively, measurements may be basedon a section of a different sample that shares the same organ type, orthe same fixation conditions, or the same disease state, or some otherproperty that recommends it as representative of the sample.

Spectra can be obtained by a human operator choosing pixel regions inthe autofluorescence image of the sample, and then obtaining the spectrafor the selected groups of pixels of each type. Alternatively, a trainedmachine classifier such as a neural network (e.g., implemented inprocessing device 216) can be used to perform the pixel regionselections, and material type-specific autofluorescence spectra can beextracted in fully automated fashion. Using the type-specificautofluorescence spectra, an S matrix can be created for each type andinverted to yield a set of S⁺ matrices to unmix pixels of each type ofsample material.

To process a sample image using type-specific autofluorescence spectra,the autofluorescence image of the sample is classified, the typeinformation associated with each pixel is used to select the appropriateS⁺ matrix for that type of sample material, and pixel is unmixed tocreate an abundance vector A. To the extent that the actual sampleautofluorescence is better matched by its type-specific spectrum than anoverall average, the resulting unmixed images will more accuratelyindicate the true sample abundances.

The foregoing procedures can be used when pixels in a sample imagecorrespond to different types of sample structures, with variations inendogenous autofluorescence that are associated with the differentstructures. Segmentation masks (or other spatial filtering techniques)can be used in conjunction with the foregoing procedures to excludeparticular sample structures that are identified from theautofluorescence image. For example, adaptive unmixing can be combinedwith the exclusion of pixels corresponding to red blood cells and/orcollagen from downstream analysis. Where red blood cells are preferablyomitted from analysis, identifying and excluding them can be beneficial.Further, adaptive unmixing can provide more accurate quantitativeinformation for surviving pixels having an autofluorescence spectrumthat differs from the average sample autofluorescence spectrum, such aspixels that correspond to stromal cells or to the extracellular matrix.

One or more autofluorescence images can also be used for otherapplications. For example, in some embodiments, an autofluorescenceimage of sample obtained according to the methods described herein canbe used to generate a synthetic H&E image (or another type of syntheticimage) of the sample. In certain embodiments, where different regions ofa sample are scanned according to a scanning pattern and the sample andbackground images are then assembled to form a larger image (e.g., awhole-slide image) of the sample, the autofluorescence spectracorresponding to different regions of the sample can be used to registersets of multispectral sample images that correspond to the differentregions. That is, the autofluorescence spectra can be used forpixel-based registration of different sets of sample images.

In circumstances where multiple samples are examined, it should beappreciated that while the methods and systems describe herein permitendogenous autofluorescence to be measured for each sample, theendogenous autofluorescence spectrum can also be measured for only asubset of the samples, or from one or more witness or reference samples,and then the measured endogenous autofluorescence spectrum can be usedin spectral unmixing operations performed on sets of sample images fromother samples, where endogeneous autofluorescence for those othersamples is not independently measured.

The methods and systems described herein are compatible with serialstaining-and-imaging protocols, and can improve detection sensitivity insuch protocols. In protocols of this type, a sample is incubated with nantibodies at once, and they are detected m at a time, by selectivelyengaging dyes and antibodies with DNA barcode technology. Theseprotocols may attempt to image n up to 30 or more, and m is 3 or 4, sothe number of imaging operations may range from 2 to 8, or even more.Other protocols can use the same broad principle, but techniques otherthan DNA barcode technology for selectively engaging the dyes andantibodies. In protocols, only m signals are typically imaged at a time,so isolating the dye signals from one another or from the counterstainis not usually too difficult. However, the measured signal levels may bemoderate or low, especially for weak species. The methods and systemsdescribed herein can be particularly useful for isolating the sampleautofluorescence that would otherwise confound detection of weakersignals. As a further benefit, the sample autofluorescence image that isobtained can be used to co-register sample images from successiveimaging sessions.

Other features and aspects of systems for obtaining sample images andmethods for analyzing and classifying spectral images are described, forexample, in the following references, the entire contents of each ofwhich are incorporated by reference herein: U.S. Pat. Nos. 8,634,607;7,555,155; 8,103,331; U.S. Patent Application Publication No. US2014/0193061; U.S. Patent Application Publication No. US 2014/0193050;and U.S. patent application Ser. No. 15/837,956.

Any of the method steps and other functions described herein can beexecuted by controller 214 (e.g., by processing device 216 of controller214) and/or one or more additional processing devices (such as computersor preprogrammed integrated circuits) executing software programs orhardware-encoded instructions. Software programs, which can be stored ona wide variety of tangible, processing device-readable storage media,including (but not limited to) optical storage media such as CD-ROM orDVD media, magnetic storage media, and/or persistent solid state storagemedia. The programs, when executed by processing device 216 (or moregenerally, by controller 214), cause the processing device (orcontroller) to perform any one or more of the control, computing, andoutput functions described herein.

In addition to processing device 216, controller 214 can also optionallyinclude other components, including a display or output unit, an inputunit (e.g., a pointing device, a voice-recognition interface, akeyboard, and other such devices), a storage unit (e.g., a persistent ornon-persistent storage unit, which can store programs and data measuredby the system, and calibration and control settings), and transmittingand receiving units for sending and receiving data and control signalsto other electronic components, including other computing devices.

EXAMPLES

(1) Lung Cancer Sample

A lung cancer sample was prepared using the reagents listed in Table 1.TSA-biotin (Akoya Biosciences, Menlo Park, Calif.) was used to stainpan-cytokeratin (see Table 1) and a final incubation with Streptavidin,Alexa Fluor™ 750 conjugate (ThermoFisher Scientific, Waltham Mass.) wasperformed at 1:200 dilution for 1 hour at room temperature.

Sample images were obtained using the system shown in FIG. 2. Fourepi-filter cubes were used in filter 206, having the optical responsesshown in FIGS. 5A-5D, respectively. Light source 202 produced light insix independently controlled wavelength bands, with an adjustablebrightness and electronic shutter capability for each band. The bandsare tabulated in Table 2.

TABLE 2 LED Center Bandwidth Channel Wavelength (FWHM) UV 385 ± 5 nm 11Violet 430 ± 5 nm 18 Blue 475 ± 5 nm 22 Yellow 550 ± 5 nm 82 Red 638 ± 5nm 18 NIR 735 ± 5 nm 32

The implementation of the system of FIG. 2 formed a compound,epi-illumination fluorescence microscope operating at infiniteconjugate. All components were selected to have high lateral resolutionand high transmission in the visible and near-infrared range (780 nm).The objective lens 208 was a Nikon 10× plan-apochromat with a focallength of 20 mm and a numerical aperture of 0.45 (Nikon USA, Melville,N.Y.). A tube lens (part of detector 212) was an apochromatic lens withfocal length 145.2 mm, and the detector was an SCMOS Flash 2.8 sensor(Hamamatsu US, Bridgewater N.J.) with a pixel size of 3.63 micronssquare. The overall magnification was 7.26, and each pixel correspondedto 0.5 microns at the sample.

The first filter cube was a triple-band filter, with 3 distinctexcitation bands and emission bands. By selecting which band of thelight source was active, each excitation band was activated separately,without exciting the other bands. Because the light source waselectronically controlled, the system was able to cycle through theexcitation bands rapidly, without mechanical motion.

Three sample images were obtained in succession, corresponding to theresponse of the sample to each of the three excitation bands.Fluorescence emission light was detected in the three emission peaksshown in FIG. 5A. Thus, sample fluorescence summed over all 3 filteremission bands was detected in each image, while only one filter bandwas excited at a time.

Then, the second filter cube was placed in the optical path. The secondfilter cube was another triple-band filter, and was used in the same wayas the first filter cube, with the light source activating the LEDscorresponding to the excitation bands in this filter cube. This providedanother three images of the sample, corresponding to the samplefluorescence in response to each of these three excitation bands, asrecorded through its 3 emission bands.

The third and fourth filter cubes were single-band filters, with oneexcitation and emission band each. These were each cycled into theoptical path, and one image was captured with each.

This provided a total of eight images of the sample, at 8 differentexcitation and emission wavelength band combinations.

Stage 210 was used to sweep out a raster pattern while taking images ofthe sample. For speed, and to reduce the number of filter-changeoperations of mechanism, the pattern executed a group of 4 lines of theraster pattern path for each filter in turn, taking images at each sitein a given row, and stepping through several rows. Then, the next filterwas engaged, and the same pattern was executed, and its images weretaken. This was continued until the group of rows had been imaged forall filters. Then, the raster pattern was continued for the next set of4 rows. In this way, the entire sample was imaged using a total of 24rows in 6 groups. Overall, the acquisition time was 4 minutes and 35seconds for a sample scan measuring 12 mm×16 mm.

As part of the overall imaging operation, controller 214 created a mapof the sample location, and measured focus at a grid of points withinthe sample region. This grid was used to set the focus mechanism of lens208 during the raster scan. Normalized variance was used as thesharpness measure to select best focus, and the focus was interpolatedfor each image in the raster using a Delaunay triangle mesh to fit theresulting surface.

Software operating on controller 214 (and specifically, processingdevice 216) was used to process each image and create a whole-slidemosaic image containing 8 spectral channels corresponding to each filterand excitation combination. The individual images acquired during theraster imaging were each corrected for shading, using a differentshading pattern for each filter and excitation setup. They were thenassembled into a mosaic based on the known pixel size at the sample, andthe raster pattern grid. This mosaic was saved as a pyramidal TIFF fileusing 512×512 pixel tiles and LZW lossless compression.

In spectral imaging, it is conventional to refer an image where everypixel contains measurements at a number of spectral bands as an imagecube. This mosaic contains image cubes at several pyramidal resolutionlevels.

Two additional steps were performed. First, the epi-filters werecharacterized by measuring the pixel shift at the sensor produced when asample was imaged with each filter in turn. Epi-filter optics willintroduce an image shift unless the dichroic and emission filterelements are perfectly free of wedge. In practice, image shifts of 2-10seconds of arc are typical even for so-called “zero-wedge” filter sets.This shift is systematic and repeatable. In the experiment beingdescribed, one pixel corresponds to 5.5 seconds of arc.

Based on the measured pixel shift at the sensor, and the known pixelsize of 0.5 microns, the stage locations were offset from the nominalvalues during the raster scan so as to oppose the wedge produced by eachfilter. Thus the stage location used for a given field in the raster wasvery slightly different when imaging with the first filter than for eachof the subsequent filters, to counter the optical shift of the epioptics. In this way, a pixel in the image corresponds to the exact samepoint in the sample despite the optical shift introduced.

Second, residual chromatic focus shift was measured for the objective,based on measuring best focus of a sample for all filter and excitationband combinations. During imaging, the focus setting was shifted by thisamount, synchronously with the epi-filter selection and LED bandselection, to compensate for this effect. If one denotes the focusdimension as the z-direction, this procedure seeks to record exactly thesame layer of the sample along the z-direction in all spectral channelsin the resulting multispectral image.

Once the image was acquired as described above, an unmixing step wasperformed to separate the spectral channels of the mosaic into estimatesof the contributions from each pure spectral contributor. As describedabove, each pure spectral contributor corresponded to one of the applieddyes (or counterstain), or to the endogenous autofluorescence of thesample.

The spectral response was modeled for each dye and counterstain takingaccount of the LED signal, the camera response, the excitation filterresponses and emission responses, and the measured excitation andemission responses for each dye and counterstain. This was done in theopen-source R programming environment (R Foundation for StatisticalComputing; Vienna, Austria). The results are shown in Table 3, as thefirst 7 data rows, corresponding to the spectral bands named B1-B7 inthat table.

TABLE 3 Spec- tral Band Dy Opal Opal Opal Opal Filter LED Name DAPI 431520 570 620 690 Cy7 206 UV B1 1.000 0.041 0.001 0.001 0.000 0.002 0.000205b Violet B2 0.013 1.000 0.014 0.005 0.000 0.004 0.002 205c Blue B30.000 0.059 1.000 0.002 0.000 0.000 0.000 205a Yellow B4 0.000 0.0000.008 1.000 0.041 0.013 0.000 205b Yellow B5 0.000 0.000 0.000 0.0311.000 0.005 0.021 205a Red B6 0.000 0.000 0.000 0.000 0.036 1.000 0.005205b NIR B7 0.000 0.000 0.000 0.000 0.002 0.021 1.000 205c Violet X0.000 0.000 0.000 0.000 0.014 0.010 0.000

FIG. 6A shows dye excitation responses for the above counterstain andsix dyes used as immuno-fluorescent labels, and FIG. 6B shows theemission responses. FIG. 6C shows the emission response ofautofluorescence in a formalin-fixed paraffin-embedded lung cancersample when excited at 387 nm, and when excited at 425 nm.

Alternatively, pure spectra (or spectral library entries) can be derivedfrom measured information. For example, to obtain a spectral libraryentry for autofluorescence, a sample with no immuno-fluorescent labelscan be imaged to obtain measured images in 8 individual spectral bands,the strength of autofluorescence signal is extracted for individualpixels, and the signal strengths are scaled by exposure time, to producea time-normalized autofluorescence spectrum following the normalizationstep.

To obtain a spectral library entry for an individual dye used inimmuno-fluorescent labeling, a sample can be prepared with primaryantibody, secondary antibody, and/or dye, using antigen retrievalsolution, blocking buffer, wash reagents and other ancillary reagents,to produce a singly stained sample. This sample is imaged to obtainmeasured images in 8 individual spectral bands, the strength of thecombined dye and autofluorescence signals is extracted for individualpixels, the signal strengths are scaled by exposure time, and thesignals are corrected for the autofluorescence contribution to obtain atime-normalized corrected dye spectrum.

To obtain a spectral library entry for a counterstain such as DAPI,where the sample is prepared with the counterstain to produce acounterstained sample, the sample is imaged to obtain measured images in8 individual bands, the strength of the combined dye or counterstainsignal plus the autofluorescence signal is extracted for individualpixels, the signal strengths are scaled by exposure time, and thesignals are corrected for the autofluorescence contribution to obtain atime-normalized corrected counterstain spectrum.

Two samples were prepared, one stained with only Opal620 and one stainedwith only Opal690. These were then imaged and the relative response wasmeasured in bands B5, B6, and X. This was used to populate the X bandrow of Table 3 for these two dyes. Measurements on singly-stained dyesamples for the other dyes showed no measurable response in the X band.

Table 3 shows the relative response of each dye for an image where allspectral bands have equal exposure times, normalized by the signal levelin the brightest band. It is often useful to scale signal counts by theexposure time and to perform calculations in units of counts per unittime, such as counts per millisecond. Signal levels in a scientificdigital camera are proportional to exposure time, and scaling byexposure time provides two practical benefits. First, it enables one toperform spectral calculations on images taken where the differentspectral bands have unequal exposure times, in exposure-scaled counts,so that the spectra are unaffected by the acquisition conditions.Second, and for the same reason, it enables one to compare spectra orperform spectral calculations involving multiple images, where theimages have different exposure times from one another. This approach wasused in present procedure.

Next, two FFPE samples which had no dyes applied were imaged using thesame apparatus and method described above. This yielded pyramidal mosaicimage cubes with spectral bands B1-B7 and X. One sample was a lungcancer section, and the other was a breast cancer section, each 4-5microns thick.

The spectra were measured at various locations corresponding torecognizable biological structures such as stroma, red blood cells,generalized extracellular matrix, collagen, and so on. This measurementwas done by taking a pixel-average of the signal in a group of pixelschosen to represent the structure of interest. This was done byexporting images from each of the individual spectral band planes fromthe pyramidal TIFF image, and then assembling the planes into an imagecube using Nuance® software from Akoya Biosciences (Waltham, Mass.).

In addition to measuring the spectra for pixel groups chosen tocorrespond to particular biological structures, a measurement was madealong an irregular path extending across a large variety of structures,to obtain a structural-average spectrum.

The resulting spectra are given in Table 4 for four types of structuresin the lung cancer tissue sample, and its average, and in Table 5 forseven types of structures in the breast cancer tissue sample, along withits average. All of these are listed in time-scaled units of counts permillisecond.

TABLE 4 Lung Cancer Sample Red Extra- Collagen Blood cellular StromaFilter LED Structures Cells matrix Cells Average 205a UV 5.017 4.1192.217 2.127 2.315 205b Violet 5.841 4.280 0.901 1.826 1.434 205c Blue1.010 0.851 0.121 0.271 0.210 205a Yellow 0.187 0.167 0.023 0.059 0.041205b Yellow 0.073 0.071 0.008 0.028 0.016 205a Red 0.038 0.040 0.0020.021 0.006 205b NIR 0.005 0.003 0.002 0.003 0.002 205d Violet 0.1810.130 0.039 0.072 0.039

TABLE 5 Breast Cancer Sample Red Tumor Collagen Blood non- Tumor StromaStroma Stroma Filter LED Structures Cells nuclear nuclear matrix cellspuncta Average 205a UV 4.083 4.019 1.680 1.429 1.736 2.155 3.126 1.702205b Violet 4.101 4.762 1.400 1.061 1.258 1.846 2.948 1.326 205c Blue0.516 0.826 0.175 0.114 0.139 0.252 0.363 0.154 205a Yellow 0.069 0.1420.032 0.020 0.023 0.045 0.049 0.026 205b Yellow 0.027 0.061 0.017 0.0120.011 0.020 0.018 0.012 205a Red 0.024 0.052 0.014 0.011 0.012 0.0160.017 0.012 205b NIR 0.003 0.007 0.003 0.004 0.003 0.004 0.004 0.003205d Violet 0.077 0.109 0.045 0.039 0.042 0.053 0.062 0.043

Based on these, spectral unmixing was performed. In a linear system ofindependent spectral contributors, the following linear algebraicequation can be used to describe the measured spectrum at any imagepixel:M=S*A  [1]

M is the measured spectrum at a given pixel, S is a matrix whose columnsare the spectra of the individual components (dye, counterstain, orautofluorescence), and A is a column vector with the abundances of thecomponents in the sample. In other words, Equation (1) states that themeasured signal M is a linear superposition of components in the samplewith spectra S, according to their abundance A.

If a pseudo-inverse S⁺ exists for S, one can left-multiply both sides ofEquation [1] by it, to obtainS ⁺ *M=S ⁺ *S*A=(S ⁺ *S)*A=I*A=A  [2a]ThusA=*M  [2b]

Equation (2b) is the central spectral unmixing equation, which enablesone to calculate the abundance vector A for the pure spectralcontributors in the sample by left-multiplying the measured spectrum Mat a given pixel by the pseudo-inverted spectral matrix S⁺.

In the present example, the goal is to unmix the measured signal intocontributions from the 6 immuno-labeling dyes, the DAPI counterstain,and the tissue autofluorescence. Accordingly, S has 8 columns,corresponding to the seven dye (or counterstain) spectra, along with anautofluorescence spectrum. The lung-cancer average spectrum, normalizedby the signal level in the brightest band, was used for theautofluorescence spectrum.

Each column had 8 entries, corresponding to bands B1-B7 and X, meaning Swas an 8×8 matrix. Because S was square, S⁺ was calculated by directinversion. More generally, S⁺ can be calculated by methods well-known inlinear algebra, such as the Moore-Penrose technique.

The resulting S⁺ matrix contained the coefficients that transform a rawspectral measurement M of a sample pixel into a vector of purecontributor abundances A, in time-scaled measurement space. It was usedto transform the raw spectral mosaic image, in which each pixelcontained measured signals M, into an unmixed mosaic image in which eachpixel contained the abundances A of the individual pure spectralcontributors (i.e., the dyes, counterstain, and the endogenousautofluorescence).

The raw sample images corresponding to bands B1-B7 and X are shown inFIGS. 7A-7H, and the unmixed pure spectral contributor images are shownin FIGS. 8A-8H. The images in FIGS. 7A-7H and 8A-8H show only a selectedregion of the sample. FIGS. 9A-9H show unmixed pure spectral contributorimages of the entire sample.

Comparison of pairs of images demonstrates the ability of the methodsand systems described herein to provide accurate quantitativeinformation about molecular targets in samples. For example, FIG. 7Bshows the raw image for B2, which is the primary spectral band whereDy430 emission was observed. This dye was a marker for CD8 in thesample. In FIG. 7B, one can see contributions associated with CD8, whichlocalized primarily in membranes of lymphocytes, specifically cytotoxicT-cells. These were evident as compact bright rings in FIG. 7B.

But there are many other structures visible in FIG. 7B that areunrelated to true CD8 localization in the sample that make this image anunreliable indicator of that type of cell. For example, there is ageneral background signal level in the extra-cellular matrix; large,bright, irregular features that appear to be collagen structures; andother cells in the tissue. Overall, the majority of the signal was dueto sources other than the CD8 label.

This interference makes it more difficult to accurately identify thepresence, number, or location of cytotoxic T-cells, and tends to degradean assay based on such one or more of these measures. Based on thelocation and shape of the confounding features, the interference arisesprimarily from endogenous tissue autofluorescence rather than from otherdyes used to label the sample.

FIG. 8B is the unmixed pure spectral contributor image associated withthe dye Dy430. Comparing this against FIG. 7B, the interfering signalsassociated with endogenous tissue are either weak or completely absent.Yet, the true CD8-associated features seen in the image of FIG. 7B arepresent without apparent reduction in strength. Also, several faint CD8ring features are observed that are difficult to locate in FIG. 7Bbecause of background or interfering signals. This illustrates at oncethe benefits of multispectral imaging and spectral unmixing as atechnique for isolating signals and removing autofluorescence.

A comparison of other raw spectral images against the unmixed componentabundance images for DAPI, Opal520, and Opal570 shows comparableeffectiveness in isolating the desired component from interference fromautofluorescence. In FIG. 8A, the DAPI pure spectral contributorabundance image, this nuclear counterstain is used to identify, segment,and count tumor and stroma cells. The DAPI image in FIG. 8A providesthis information essentially free from unwanted other content, whereasthe raw spectral in FIG. 7A contains many small, bright structures thatdo not correspond to DAPI stain. These structures appear to arise fromendogenous tissue autofluorescence. The spectral image shown in FIG. 7Acorresponds to a narrow-band excitation and emission filter that werechosen to be optimal for fluorescent imaging of DAPI. To the extent thatthe filter selection was indeed optimal, the signal-to-noise shown inthis image—its ability to discern the desired DAPI signal without otherconfounding signals—illustrates the best performance that conventionalwhole-slide imaging can achieve.

Similarly, Opal570 was used to label FoxP3 proteins in the sample, whichtend to localize in nuclei of regulatory T-cells (e.g., “Treg” cells).The unmixed pure spectral contributor image in FIG. 8D (corresponding toOpal570) shows clean labeling of nuclei. This indicated that theimmuno-fluorescent labeling of FoxP3 achieved high specificity, so theOpal570 label appears to be localized to the intended structures.Brightly-labeled Treg cells show 150-250 counts, and fainter cellsexpressing 50 counts are easily detectable against a low background of10 counts or less. In contrast, the raw sample image in FIG. 7D has muchmore general background, with many kinds of structures having signallevels of 25 counts that do not correspond to true FoxP3.

Next, consider FIG. 7H, which is the raw sample image at band X. Theexcitation filter and emission filter for this band were chosen todetect endogenous autofluorescence from FFPE samples, rather than anycounterstain or dye. Further, they were chosen to be as insensitive aspossible to the counterstain and dye(s) applied to the sample, so it ispossible to obtain an image of endogenous autofluorescence in a sampledespite the presence of 4 or more dyes. In general, multispectral sampleimaging protocols do not include selecting excitation and emissionwavelength bands with these properties. To the contrary, conventionalmethods for obtaining an image of endogenous autofluorescence in such asample involves spectrally unmixing a multispectral image cube where oneof the spectra is autofluorescence, and looking at the autofluorescenceabundance image. In such cases, the raw spectral cube typically includesa high number of spectral channels to separate the autofluorescencesignal from the other fluorescent signals in the sample. This greatlyincreases computational time and data storage, making it impractical formany uses, such as whole-slide scanning or digital pathology workflows.

In contrast, band X is only weakly affected by other signals,particularly due to non-endogenous dyes and counterstains. This makes itmuch easier to obtain an accurate estimate of endogenous sampleautofluorescence without a high number of spectral channels. Forexample, in some circumstances, only one spectral channel (i.e., asingle wavelength measurement) is used for each dye and counterstain,plus one for autofluorescence.

FIG. 8H is the unmixed abundance image for autofluorescence. The imagelooks broadly similar to FIG. 7H, which is not surprising since the Xspectral band was designed to respond only weakly to signals other thanendogenous autofluorescence. Both images show features corresponding tocollagen structures, generalized extracellular matrix, stroma structuresincluding cell nuclei, and red blood cells.

However, there are important differences. The raw spectral image shownin FIG. 7H includes some signal associated with the Opal620-labeled CD68targets, and Opal690-labeled PD1 targets, which are largely or entirelyabsent in the unmixed autofluorescence abundance image of FIG. 8H. Thiscan be understood in terms of the dye spectra described above. The Xband shows 1.4% response for Opal620 dye, and 1.0% response for Opal690dye, compared with the time-normalized response in the band where eachdye responds most strongly.

Thus, the raw spectral image in FIG. 7H includes some signal that isassociated with the Opal620 component, and some that is associated withthe Opal690 component. Spectral unmixing produces the autofluorescenceimage in FIG. 8H that more correctly reveals the true endogenousautofluorescence signal. In this particular example, the amount ofdifference is relatively modest.

In some embodiments, the autofluorescence image can be analyzed toidentify particular tissue structures, cells, or regions that are eitherignored or treated specially in downstream image analysis. In thediscussion of FIG. 8D above, in regions associated with red blood cells,the raw spectral image of FIG. 7D showed signal that did not properlybelong to that dye component. Without wishing to be bound by theory, apossible explanation is that the autofluorescence for red blood cellsdiffers from that of the average sample structure. It is valuable torecognize that these are red blood cells rather than Treg cells, despitesome signal being present in the FoxP3 abundance image. Accordingly, insome embodiments, red blood cells can be identified based on theautofluorescence image, and marked as red blood cells for downstreamanalysis, ensuring they will not be mistaken for FoxP3-positive nuclei.

This step can be used in other analytical procedures as well. Forexample, in certain embodiments, the exact consequence of red blood cellfluorescent emissions could be different from the consequences here,although it still may be beneficial to recognize and segregate red bloodcells. In some embodiments, structures associated with collagen mayconfound or degrade a particular measurement, and regions of the sampleassociated with collage structures can be identified from theautofluorescence image and eliminated from the analytical procedure, orotherwise analyzed according to different criteria or algorithms.

It should be appreciated that various methods are available to decomposethe sample images. In particular, the spectral unmixing calculation canbe performed in a variety of ways, including by changing the order inwhich certain steps are performed, and omitting certain steps entirely.As an example, it is not necessary to unmix all members of the abundancematrix to make use of any one unmixed component. Specific rows of theunmixing matrix can be selectively used for the unmixing proceduredepending upon which pure spectral contributors are being elucidated.

Further, because the methods and systems described herein directlyobtain a background image of the sample (which correspondsapproximately—or even almost identically—to a sample autofluorescenceimage), the sample autofluorescence image can be obtained (e.g., fromthe background image) prior to unmixing the sample images to obtain thepure spectral contributor images. This two-step analysis procedure canbe particularly useful when performing adaptive spectral unmixing thatdepends on the nature of the autofluorescence spectrum at specificsample locations, and/or when pixels corresponding to certain samplestructures (such as red blood cells and/or collagen) are identified fromthe autofluorescence image and marked, e.g., to exclude those pixelsfrom further analysis.

To obtain the autofluorescence image ahead of the pure spectralcontributor images, the autofluorescence abundance can simply be unmixedfirst in a two-step unmixing procedure. Contributions fromnon-endogenous spectral contributors (e.g., the applied dyes) are thenunmixed based on the sample images corresponding to the raw spectralbands (e.g., bands B1-B7 in this example) and the autofluorescenceabundance A_(x), rather than using the background image corresponding toband X. The unmixing matrix coefficients are revised to take account ofthe use of A_(x), as follows:A _(x) =S ⁺ _(x) *M  [3]

Here S⁺ _(x) refers to row x of the inverted spectral matrix S⁺. Theother rows of the unmixing matrix can also be revised to take account ofthe contributions introduced via A_(x), and the result can be a morediagonal matrix. Because the matrix is more diagonal, it may be possibleto omit small-valued terms, speeding up the calculation, without a largeimpact on accuracy.

(2) Cell Counting

To investigate cell counting based on the sample autofluorescence image,two serial sections of a lung cancer sample were prepared. One sectionwas an unstained negative control prepared without any dye deposition.The other section was stained with a multiplex panel using reagentslisted in Table 6, followed by a final incubation with Opal Polaris 780Anti-Dig (Akoya Biosciences, Waltham, Mass.) at 1:25 dilution for 1 hourat room temperature. All other incubation times were the same as thosefrom the prior example. Sample images were obtained using the samesystem described in the prior example.

TABLE 6 Primary Primary Antibody Antibody Tyramide Staining Vendor,Catalog #, Dilution Dilution Order Target Clone, Species Factor TyramideReagent Factor 1 PD-L1 CST (Danvers 1:300 Opal 520 (Akoya 1:150 MA),13684, Biosciences, Menlo E1L3N, a-Rb Park, CA) 2 FoxP3 Abcam 1:100 Opal570 (Akoya 1:300 (Cambridge, UK), Biosciences, Menlo ab20034, 236A/E7,Park, CA) a-Ms 3 PD-1 AbCam 1:300 Opal 690 (Akoya 1:100 (Cambridge, UK),Biosciences, Menlo ab137132, Park, CA) EPR4877(2), a-Rb 4 CD68 Dako(Santa Clara 1:100 Opal 620 (Akoya 1:150 CA), M0876, PG- Biosciences,Menlo M1, a-Ms Park, CA) 5 CD8 AbD Serotec 1:300 Opal Polaris 480 1:150(Oxford UK), (Akoya Biosciences, MCA1817, 4B11, Menlo Park, CA) a-Ms 6Pan- Novus (Littleton 1:200 Opal TSA-Dig 1:100 Cytokeratin CO),NBP2-29429, (Akoya Biosciences, AE1/AE2, a-Ms Menlo Park, CA)

Following measurement of the background image (which was assumed tocorrespond to the sample autofluorescence image), a pixel-basedclassification algorithm was trained to identify red blood cells usingthe automated tissue segmentation tool in inForm® software (AkoyaBiosciences, Menlo Park, Calif.). Only the autofluorescence image wasused as an input into the classifier. FIG. 10A shows a set of imagesderived from the sample prepared with the dyes shown in Table 6, whileFIG. 10B shows a set of images derived from the negative control sample(with no dyes). Images 901 in FIGS. 10A and 10B show the respectiveautofluorescence images of the samples.

To train the machine learning algorithm, regions were manually drawn onthe negative control autofluorescence image to identify pixels that wereeither positive or negative for red blood cells. Results from thetrained algorithm for both sections are binary classification maskscorresponding to images 902 in FIGS. 10A and 10B.

Separately, a cell segmentation algorithm was developed to identifyFoxP3-positive cells using the adaptive cell segmentation tool in theinForm® software. Only the unmixed Opal 570/FoxP3 abundance image(images 903 in FIGS. 10A and 10B) was used as an input to thissegmentation algorithm. Thresholds for segmentation were set to identifyobjects in the image that have Opal 570 signal and the elliptical shapeof a cell nucleus (images 904 in FIGS. 10A and 10B). All cellsidentified in the negative control sample (image 904 in FIG. 10B) werefalse positive detections of autofluorescent species such as red bloodcells that have similar morphology to cell nuclei.

Finally, the masks from both algorithms (images 902 and 904 in FIGS. 10Aand 10B) were combined to retain only the FoxP3-positive cells that donot overlap with any pixels that were identified to be red blood cellsin panels. The resulting masks (images 905 in FIGS. 10A and 10B) show areduction in the number of false positives relative to images 904.

What is claimed is:
 1. A method, comprising: exposing a plurality ofnon-endogenous spectral contributors in a biological sample toillumination light and measuring light emission from the sample toobtain N sample images, wherein each sample image corresponds to adifferent combination of a wavelength band of the illumination light andone or more wavelength bands of the light emission, wherein multiplewavelength bands of the light emission from the plurality ofnon-endogenous spectral contributors define a wavelength range, andwherein N>1; and exposing the sample to illumination light in abackground excitation band and measuring light emission from the samplein a background spectral band to obtain a background image of thesample, wherein the background spectral band comprises a distribution ofwavelengths having a full width at half maximum (FWHM) spectral width Δλand a center wavelength λ_(c) within the wavelength range, and whereinthe wavelengths within the background spectral band correspond towavelengths within a range from λ_(c)−Δλ/2 to λ_(c)+Δλ/2, wherein foreach of the plurality of non-endogenous spectral contributors in thesample exposed to the illumination light in the background excitationband, a spectral emission intensity at each wavelength within thebackground spectral band is 10% or less of a maximum measured spectralemission intensity of the non-endogenous spectral contributor followingexcitation of the non-endogenous spectral contributor in each of thewavelength bands of the illumination light and the background excitationband.
 2. The method of claim 1, further comprising obtaining anautofluorescence image of the sample from the background image.
 3. Themethod of claim 2, further comprising displaying the autofluorescenceimage on a display device.
 4. The method of claim 2, further comprisingdetermining, at each of multiple locations in the sample, an amount ofautofluorescence emission from the sample.
 5. The method of claim 4,further comprising, at each of the multiple locations in the sample, andfor one or more of the N sample images: adjusting values correspondingto sample emission intensity to correct for autofluorescence emissionfrom the sample based on the amount of autofluorescence emission at eachof the multiple locations and at least one pure spectrum ofautofluorescence emission from the sample.
 6. The method of claim 5,wherein the at least one pure spectrum of autofluorescence emissioncomprises multiple pure spectra of autofluorescence emission, andwherein the multiple pure spectra of autofluorescence emission eachcorrespond to a different subset of the multiple locations.
 7. Themethod of claim 6, further comprising: decomposing at least some of theN sample images based on the amount of autofluorescence emission fromthe sample at each of the multiple locations to obtain M spectralcontributor images, wherein each of the M spectral contributor imagescorresponds to light emission only from a different one of thenon-endogenous spectral contributors; and at each of the multiplelocations, determining an amount of the M non-endogenous spectralcontributors in the sample.
 8. The method of claim 7, further comprisingdecomposing the at least some of the N sample images based on at leastone pure spectrum of autofluorescence emission from the sample.
 9. Themethod of claim 8, wherein the at least one pure spectrum ofautofluorescence emission comprises multiple pure spectra ofautofluorescence emission, and wherein the multiple pure spectra ofautofluorescence emission each correspond to a different subset of themultiple locations.
 10. The method of claim 2, further comprisingclassifying pixels of one or more of the sample images into differentclasses based on information derived from the autofluorescence image.11. The method of claim 10, wherein the different classes correspond todifferent cell types in the sample.
 12. The method of claim 1, whereinfor each of the plurality of non-endogenous spectral contributors in thesample exposed to the illumination light in the background excitationband, the spectral emission intensity at each wavelength within thebackground spectral band is 4% or less of the maximum measured spectralemission intensity of the non-endogenous spectral contributor followingexcitation of the sample in each of the wavelength bands of theillumination light and the background excitation band.
 13. The method ofclaim 1, wherein for each of the plurality of non-endogenous spectralcontributors in the sample exposed to the illumination light in thebackground excitation band, the spectral emission intensity at eachwavelength within the background spectral band is 2% or less of themaximum measured spectral emission intensity of the non-endogenousspectral contributor following excitation of the sample in each of thewavelength bands of the illumination light and the background excitationband.
 14. The method of claim 1, wherein N>3.
 15. The method of claim 1,wherein N>5.
 16. The method of claim 1, wherein the sample comprises Mnon-endogenous spectral contributors, and wherein M≤N.
 17. The method ofclaim 16, wherein M>4.
 18. The method of claim 17, wherein a sum ofspectral emission intensities of each non-endogenous spectralcontributor in the sample at each wavelength within the backgroundspectral band is 10% or less of a total fluorescence emission intensityin the background spectral band.
 19. The method of claim 16, whereinM>6.
 20. The method of claim 16, wherein the M non-endogenous spectralcontributors comprise one or more fluorescent species that selectivelybind to different chemical moieties in the sample.
 21. The method ofclaim 20, wherein the one or more fluorescent species comprise one ormore immunofluorescent probes.
 22. The method of claim 20, wherein the Mnon-endogenous spectral contributors comprise one or more counterstains.